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Summary 


Based on theories of adult learning and airline industry guidelines for Crew Resource Management 
(CRM), the stated objective during Line Oriented Flight Training (LOFT) debriefings is for 
instructor pilots (IPs) to facilitate crew self-analysis of performance. This study reviews 19 
LOFT debriefings from 2 major U.S. airlines to examine the relationship between IP efforts at 
facilitation and associated characteristics of crew participation. A subjective rating scale called the 
Debriefing Assessment Battery was developed and utilized to evaluate the effectiveness of IP 
facilitation and the quality of crew participation. The results indicate that IP content, 
encouragement, and questioning techniques are highly and significantly correlated with, and can 
therefore predict, the degree and depth of crew participation. 
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1. Introduction 

This study evolved from a larger project in which we are studying the content and conduct of 
Line Oriented Flight Training (LOFT) debriefings (Dismukes, McDonnell, Jobe, & Gregorich, 

1 995). Most of the major U.S. airlines use LOFT as the key training tool in their ongoing Crew 
Resource Management (CRM) training programs, and the pilots in most major U.S. airlines are 
required to go through recurrent LOFT training every year. Helmreich and Foushee (1993) assert 
that in order for the LOFT training to be meaningful, it is essential to debrief after each session. At 
all of the airlines studied, the organizations' stated objective for LOFT instructors is to act as 
facilitators, enabling the crew members to perform their own analysis of their performance. 
Unfortunately, LOFT instructors are given very little guidance on what facilitation is or about how 
to facilitate and this contributes to the common problem that debriefing sessions generally don't 
reach their full potential (Chidester, 1993; see also Butler, 1993; Helmreich, Chidester, Foushee, 
Gregorich, & Wilhelm, 1990). 

Although it is widely assumed that facilitation is the best debriefing strategy, there does not 
appear to be much empirical research on how facilitation of group interaction can enhance group 
performance (Hirokawa & Gouran, 1989) or on specific techniques conducive to facilitation which 
may be applicable to the debriefing of airline crews. Most of the literature found and cited here 
consists of assertions for which no empirical evidence is presented. Another problem is that most 
of the evidence that is available does not necessarily generalize to the LOFT environment because 
there are important differences between instruction in commercial aviation and that in the 
educational setting, from which most of the research comes (Reynolds, 1992; Telfer, 1993). 
Consequently, this study will review actual LOFT debriefings in order to examine the relationship 
between IP efforts at facilitation and associated characteristics of crew participation. 

1.1 Crew Resource Management 

Crew Resource Management was defined by Hawkins (1987) as the management and 
utilization of all available resources on the aircraft, including people, equipment, and information. 

The underlying philosophy of CRM training is that cockpit performance can be improved through a 
heightened awareness of CRM skills (Smith, 1994), such as communication, coordination, and task 
and workload management. The ultimate goal is to increase the safety, effectiveness, and efficiency 
of flight operations (Gregorich & Wilhelm, 1993). According to the Federal Aviation 
Administration (FAA) this goal is achieved through the integration of technical proficiency and the 
human skill of crew coordination (Driskell & Adams, 1992). 

CRM training has been utilized since 1979 when United Airlines developed the first formal 
CRM training program (Smith, 1994). As a precursor to formal CRM training, NASA and 
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Northwest Airlines began to develop the training of airline pilots as crews by about 1975 
(Hawkins, 1987). The idea of crew resource management evolved from studies conducted by NASA 
researchers in the 1970's that attempted to determine the factors underlying air carrier accidents. 

The FAA reports that interviews with airline pilots indicated that a consistent component in 
accidents was what pilots perceived as inadequate training in crew coordination (Driskell & Adams, 
1992). This finding led to research such as Ruffell-Smith's (1 979) simulation study which found 
that crew coordination was significantly related to safety. Specifically, most problems were related 
not to technical knowledge and skills, but to breakdowns in crew coordination (Smith, 1994). 

Statistics from 1959-1989 on the causes of airline accidents indicate that over 70% of worldwide 
public transport accidents were due to flight crew error (Helmreich & Foushee, 1993). Billings 
and Reynard (1981) found that over 70% of a large group of incidents they analyzed that were 
anonymously reported to the Aviation Safety Reporting System (ASRS) contained evidence of errors 
in information transfer, the most common (37% of the incidents) being failure to initiate 
communication. 

Findings such as those above led to the 1979 NASA/Industry Workshop on Resource 
Management on the Flight Deck, which converged the efforts of NASA , the military, and the 
commercial airline industry to address these issues (Driskell & Adams, 1992). The FAA also 
recognized the value of CRM training and changed the Federal Aviation Regulations (FARs) regarding 
crew proficiency training, thereby approving CRM/LOFT training sessions as a substitute for 
certain crew proficiency checks (Driskell & Adams, 1 992; Hawkins, 1 987). The FAA further 
projects that CRM may one day be a requirement in all aircrew certification programs (Driskell & 
Adams, 1992). 

1.2 Line-Oriented Flight Training 

LOFT is a component of CRM training in which pilots teamed together as flight crews fly 
realistic real-time scenarios in a mission-oriented simulator and use CRM principles to complete 
the flight successfully (Foushee & Helmreich, 1988; Smith, 1994). The development of LOFT is 
credited by Hawkins (1987) to the realization that it is not sufficient to look at individual 
proficiency alone for skills that need to be practiced in group activities (Smith, 1994). In the 
cockpit, overall performance is the outcome of the interaction of crew members, including their 
communication, coordination, and workload management (R. K. Dismukes, personal communication, 
September 19, 1995); therefore, team proficiency needs to be assessed. 

The purpose of the LOFT in CRM training is to allow flight crews to work together as teams 
to solve problems that occur (Hawkins, 1987). In the LOFT, as human and equipment errors occur 
there is no intervention. They are allowed to occur just as they are on a real flight, and it is up to 
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the crew to manage their resources effectively to resolve the errors. The goal of the LOFT is to 
improve the effectiveness of crews in managing their resources on the flight deck, including the 
management of human error (Hawkins, 1987). 

1 .3 Debriefing the LOFT 

An essential element of the LOFT session is the debriefing. Debriefing is defined by 
Thiagarajan (1986) as "the process of helping people reflect on their experiences to derive 
meaningful lessons." According to Helmreich and Foushee (1993) the debriefing of LOFT sessions 
is critical to achieving an impact on flight crew behavior. In order for the debrief to be most 
effective, some authors recommend that the discussion be led by the crew themselves using the 
instructor/facilitator and the videotape of the LOFT session as resources for the self-critique 
(Butler, 1993; Hawkins, 1987; Smith, 1994). The self-analysis of the LOFT which results from 
a crew-led discussion has been asserted by Butler to be highly effective in improving the CRM 
performance of airline pilots. 

The assumption that self-analysis is the most desirable and successful debriefing strategy is 
supported by Duval and Wicklund's (1972) Theory of Objective Self-Awareness. This theory 
proposes that objective appraisals of oneself are often forced by self-focusing stimuli, and that 
such appraisals may lead to attitude and behavioral changes (Smith, 1994). From this theory, it 
logically follows that since the goal of CRM training is to reduce human errors in the cockpit, CRM 
skills should be explored by the participants themselves rather than lectured by an instructor. 

Such objective self analysis enables pilots to examine their own management skills and to try other 
CRM techniques that may be more effective than the ones they currently use (Butler, 1991). The 
hope is that by debriefing themselves during LOFT, pilots will develop this important skill and 
begin to debrief themselves with increasing frequency on the line (Butler, 1993; Smith, 1994). 

The LOFT facilitator. As indicated above, the LOFT facilitator is one of the two key 
resources crews need to utilize to debrief themselves. Because the debriefing process is truly self- 
learning, the CRM instructor is commonly referred to as the “facilitator" (Byrnes & Black, 

1993). In the role of facilitator, the Instructor Pilot (IP), is expected to guide crews to self- 
realization and reinforce instances of effective team behavior, not lecture to them about deficiencies 
or errors that may have occurred (Helmreich, 1987; Helmreich & Foushee, 1993). According to 
the original LOFT guidelines, the LOFT instructor is not really a teacher, but a moderator whose 
role is to do everything possible to foster self-analysis and to guide the debriefing session so the 
crew can explore the full range of potential solutions to the problems encountered (Lauber & 
Foushee, 1981). Gibb (1982, p. 16) asserts that the basic philosophy underlying facilitative 
training is that there's no such thing as teaching, only the facilitation of learning, and that 
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"involvement, accountability, and feedback are essential elements of effective adult learning." 

Thus, participants are expected to accept responsibility for their own learning, and facilitators are 
expected to recognize that the active involvement of the participants is essential (Gibb, 1982). 

In its handbook on LOFT facilitation techniques, Continental Airlines (1992) has presented a 
clear and useful hierarchy for the effective facilitator based on the concepts of discovery and 
ownership. According to Continental Airlines, the facilitator's preference is always to have the 
trainees recognize what they did well and what they need to improve (discovery), and to make a 
commitment to continue and/or begin using desired behaviors and to stop using undesirable ones 
(ownership). At the top of the hierarchy is "they see it, they say it." This is the ideal in which the 
trainees recognize and analyze their own performance. In the middle is "you help them see it, they 
say it." If the trainees are not able to recognize where they did well and where they can improve, 
the facilitator can lead them to self-analysis through questioning. Finally, at the bottom of the 
hierarchy is "you help them see it, you help them say it." When the crew is unable to recognize or 
evaluate their performance the facilitator must evaluate for them to ensure that they understand 
what went well or poorly, and why. 

The videotape. In addition to the facilitator, the videotape of the LOFT session is an 
important resource for the crew to use to enhance the debriefing. The facilitator is expected to 
mark relevant events on the tape during the LOFT session so he or she can draw the crew's attention 
to important points for discussion (Mellor, 1993). R. K. Dismukes (personal communication, 
September 1 9, 1 995) asserts that a primary contribution of videos may be to remind crews what 
transpired at critical segments during the LOFT, because it is often difficult for participants to 
remember specific details of the typically long and stressful sessions. Although the effectiveness of 
video use in LOFT debriefings has not been studied empirically, the use of videos is assumed to be 
valuable because videos enable crews to review their actions and then examine and gain insight into 
their own behavior with the detachment of observers (Helmreich, 1987; Helmreich & Foushee, 

1993). Nelson-Jones (1992) reports that videos can encourage self-assessment by participants 
by helping them identify behaviors that were both helpful and harmful to the final outcome. The 
realistic feedback that videos provide can make it easier for facilitators to involve the entire crew 
in a discussion of their team performance, thus enhancing the overall potential of the training 
session (Orlady, 1993). 
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1.4 Adult Learning Theory 

The industry guidelines for facilitation are derived largely from adult learning theory. 

According to Cornwell (1979), most experts agree that adult learning is self-directed, and thus 
adults learn more in guided discussion than they do from lectures. Recent research by Smith 
(1994) has gone a step closer to demonstrating that the benefits of guided discussion may apply to 
the LOFT environment. Smith discovered that guided self-analysis produced strong gains in ratings 
on the Line/LOS (Line Operational Simulations) checklist, which evaluates performance of CRM 
skills both on the line and in the simulator, for a single experimental crew consisting of two 
undergraduate students in a CRM training course. 

A body of research based largely on survey studies of what adults dislike in the classroom 
and what constitutes good and bad meetings has resulted in several implications for training which 
have been adopted by the airline industry (Zemke & Zemke, 1981). They include the findings that 
adults strongly dislike long lectures; that their self-esteem is on the line because they tend to take 
things personally; that they can learn well, and more, from discussions with their peers; that new 
knowledge must be integrated with old through active participation; and that it is critical to 
articulate and clarify all expectations up front (Continental Airlines, 1992; Zemke & Zemke, 

1981). 

1.5 Level of Mastery 

For LOFT crews to achieve their main debriefing objective of analyzing the situation and 
evaluating their performance in depth, it is necessary to utilize a high level of cognitive processing. 
According to B. S. Bloom (1956) there are six levels of mastery, which are arranged 
hierarchically by the level of mental complexity involved in doing them (Downing, 1995). The six 
levels from least to most demanding are knowledge, comprehension, application, analysis, 
synthesis, and evaluation (Bloom, 1956; Downing, 1995). It can be assumed that pilots involved 
in recurrent training have already mastered knowledge and comprehension. Participation in the 
LOFT is an example of application; in new situations crews are expected to apply what they have 
learned. The debriefing provides the opportunity for crews to take mastery to the higher levels. 

The stated debriefing objective of having the crews analyze and evaluate their LOFT performance is 
consistent with Bloom's taxonomy of achieving the highest level of mastery. The emphasis of 
analysis is for participants to break down the overall scenario into individual parts and explore the 
relationship between the parts (Bloom, 1956; Downing, 1995). In the LOFT setting crews explore 
the relationship between events in the scenario and specific CRM topics and how these affect the 
outcome. Evaluation requires making judgments. In the LOFT setting crews are expected to evaluate 
their performance and explore ways they can improve. 
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1.6 Facilitation 

According to Gow and Kember (1 993) facilitation is oriented toward helping individuals 
develop problem solving and critical thinking skills. Facilitation is the foundation of many 
commonly used counseling techniques, several of which may provide insight into potentially 
effective LOFT facilitation techniques. To encourage participation and to influence content, Hackney 
and Nye (1973) recommend avoiding interrupting participants, which can undermine their 
participation; using silence to make it clear that the participants are responsible for the 
discussion; using active listening, which demonstrates that the facilitator is listening and provides 
"significant reinforcing value" (p. 58); and using probing questions, which prevent yes/no and 
other minimal responses. Additional facilitation techniques recommended in the counseling 
literature include letting participants know at the beginning that participation is expected, 
encouraging participants to talk to each other (interaction), and inviting participation by asking 
for general input (Nelson-Jones, 1992). 

The above counseling strategies, as well as the adult learning literature, suggest several 
techniques that can be utilized during LOFT debriefings to facilitate active and in-depth 
participation. Particularly relevant techniques include the introduction, active listening, 
questions, and silence. 

Introductions. Introductions are important because clarifying the role of the facilitator 
(Casey, Roberts, & Salaman, 1992) and indicating that participation is expected (Nelson-Jones, 
1992) are reported to set both the trainer and the group up for success (Gibb, 1982). According 
to the literature, adults need to know what is expected of them, so it is critical to clarify the 
expectations of both the trainer and the trainees before training begins (Continental Airlines, 

1992; Zemke & Zemke, 1981). 

Active listening. While facilitators need good speaking skills to be effective, good 
listening skills are also critical (Gibb, 1982). According to Gibb, active listening is one of the 
most useful tools the facilitator can utilize to encourage continued participation; it shows the crew 
that the facilitator is listening and understands what they are saying. Active listening can range 
from a simple “uh, huh” or “okay” to echoing or reflecting in one’s own words what the 
participants say. 

Questions. According to the Socratic method, learning is facilitated by questioning, 
encouraging exploration, and pushing for explanation; not by lecturing and telling the answers 
(Casey et al., 1992). Various types of questions that can aid self-assessment include “Can you give 
me a specific example?" “How did you and the other person actually behave?” and “What were 
your thoughts in the situation?” (Nelson-Jones, 1992). There are general guidelines for effective 
questioning that are recognized as standard, particularly in educational and counseling settings, but 
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few clearly defined and exemplified techniques. Mills and Roberts (1981) assert that ideally, 
questions should be brief; open (i.e. non-restrictive, don’t imply opinion or judgment); and begin 
with who, where, and when for factual responses, or what, how, and why for more in-depth and 
detailed answers. 

The use of probing questions can be particularly effective in encouraging active and in-depth 
participation, which is the main goal of the LOFT debriefing. Successful probing has been found to 
result in increased depth of discussion by getting participants to further explain and justify their 
responses (Jacobsen, Eggen, & Kauchak, 1989). Mills and Roberts (1981) established seven 
types of probes which can be used to encourage continued participation. The seven types are non- 
verbal (e.g. a nod); short verbal ("Uh, huh?"); "W" words (especially what, how, and why); 
statements like "Tell me more."; echoing of participant words; reflection of what the participant 
said with different words but the same meaning; and specialized reflections that imply more than 
stated by the participant (Eitington, 1986; Mills & Roberts, 1981). 

Silence. When questions are not answered immediately, a three to four second wait-time 
has been found to improve the quality of responses (Rowe, 1986; Jacobsen, et al., 1989). Though 
periods of silence may seem awkward, there are numerous beneficial effects of increasing wait- 
time to three or four seconds from the usual one second wait-time. Not only do the length and 
confidence of responses increase, but initiation in the form of unsolicited responses, interactive 
participant responses, and participant questions increases. In addition, responses from slower 
participants increase, speculative responses and evidence-inference statements increase, and 
failures to respond decline (Ornstein, 1990; Rowe, 1974). 

1.7 Industry Guidance 

According to the FAA (Driskell & Adams, 1992), there is no one right way to implement 
CRM training. Each of the airlines develops its own program to match its own needs and 
organizational culture. However, there are several fundamental principles of CRM which are 
essential to effective training programs. First, effective performance depends on a combination of 
technical and interpersonal skills, so both technical and CRM skills need to be included in the 
overall training curriculum. Second, CRM focuses on the attitudes and behaviors of crew members, 
and a primary focus of CRM is to develop effective team coordination; therefore CRM must include 
the entire flight crew to be effective. Finally, to acquire effective CRM skills, the active 
participation of all crew members is required. Active participation and practice, including LOFT 
simulations, are considered essential to gaining effective resource management skills (Driskell & 
Adams, 1992, pp. 8-9). 
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2. Purpose and Rationale 

The purpose of the larger study of which this study is a part was to determine (a) the extent 
to which what the crews actually do during LOFT debriefings is consistent with the airline industry 
objectives of active participation and in depth analysis, (b) whether the techniques the instructors 
use are consistent with the objective of facilitating these crew objectives, and (c) how effective the 
facilitation techniques used by the instructors are at eliciting active crew participation and in depth 
analysis of LOFT performance. The overriding goal of the larger study was to develop guidelines for 
facilitation, incorporating recommendations for specific techniques with examples of how to apply 
them, that the airlines can use to teach their instructors how to be effective facilitators. 

The main focus of the present study is to develop a tool to assess the effectiveness of IP 
facilitation and to determine whether this tool can be used to predict the desired aspects of crew 
performance, namely active participation and in-depth analysis. This tool should enable us to 
identify specific instructor techniques that correspond with active crew participation and in depth 
analysis of LOFT performance. 
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3. Method 


3.1 Subjects 

Archival data from nineteen LOFT debriefing sessions, composed of all male airline pilots 
assembled for recurrent training at two major U.S. airlines, were used in this study. All sessions 
were led by one to two IPs, and the crews consisted of line pilots assembled as either two or three 
person crews. All 19 crews included line pilots acting as a Captain (CA), and a First Officer (FO). 
Three person crews also included a line pilot acting as a Flight Engineer (FE), or, if a line pilot was 
not available, an instructor playing the role of the Flight Engineer (FEI) in both the LOFT and the 
debriefing. Specifically, eleven of the sessions included one IP, a CA, and an FO; one session 
included two IPs, a CA, and an FO; one session included one IP, a CA, 

an FO, and an FE; two sessions included one IP and one FEI, a CA, and an FO; and four sessions 
included two IPs, a CA, an FO, and an FE. These groups were formed randomly, and in this respect 
the training is representative of line operations, in which crews are newly formed each month. 

3.2 Materials 

Audiotapes and transcripts. Audiotapes and matching transcripts of real LOFT 
debriefings collected and produced in the study by Dismukes et al. (1995) were utilized for this 
study (see Appendix A). The identities of individuals and airlines involved in the study were coded 
to maintain confidentiality. 

Debriefing Assessment Battery. The Debriefing Assessment Battery is a subjective 
rating scale for use by subject matter experts. The battery was developed for this study to 
characterize IP effectiveness at facilitation and the nature of crew participation in LOFT debriefings 
in a systematic way. The Debriefing Assessment Battery is based on a combination of adult learning 
and facilitation literature, existing rating scales by M. M. Connors (1995) and R. H. Moos 
(1994), face valid assumptions of what constitutes good facilitation, and the airline industry's 
guidance to their instructors on how to facilitate LOFT debriefings. The battery incorporates a 
seven-point Likert scale ranging from poor (1) to outstanding (7). The purpose of the scale is to 
evaluate IP performance on several dimensions central to facilitation, to evaluate the extent and 
character of crew participation, and to explore the relationship between what the IP does and what 
the crew members do. Appendix B presents the complete Debriefing Assessment Battery. 

The Debriefing Assessment Battery contains 28 items grouped into seven composite 
categories consisting of four items each. Five of the categories rate the IP while the remaining two 
rate the crew. The five IP categories are introduction (letting the crew members know what is 
expected), questions (to focus on topics and provoke the crew to participate deeply), encouragement 
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(the degree to which the IP encourages and enables the crew to participate actively and deeply), 
content of discussion (getting the crew to analyze and evaluate their own CRM performance), and 
videos (to remind the crew of what happened and enable them to evaluate their performance 
objectively). The two crew categories are content of discussion (the depth to which they analyze 
and evaluate the situation and their performance) and activity (how actively and thoroughly they 
participate in and initiate discussion). 

3.3 Procedure 

Two expert raters independently rated the IPs and crews from each of the 19 debriefing 
sessions using the Debriefing Assessment Battery (see Appendix B). The raters listened to each 
LOFT session audiotape while reading the verbatim transcripts to preserve the subtle effects of 
vocal inflection and other nuances of verbal communication that are essential to accurate 
interpretation. Both raters took notes while they reviewed the sessions to assist in scoring the 
battery and support the derived ratings. For each of the first 10 sessions, after independently 
completing the battery, the raters compared and discussed their ratings on the individual battery 
items before going on to the next session. During each discussion, if either of the raters believed 
their ratings needed to be changed based on issues raised by the other, they revised their scores 
accordingly, although no effort was made to reach consensus on each item. This process was 
implemented in order to resolve any discrepancies in the interpretation of events that transpired 
during the sessions or the meaning of the battery items. For the final nine sessions the raters 
completed the batteries independently without discussion. For all 19 crews the final individual and 
composite scores were obtained by averaging the raters' scores. 

The composite scores for the five IP and two crew categories were calculated by adding the 
combined raters' scores for the four items in each category. For videos, however, the composite 
score was calculated for only three of the four items because the fourth item was added to the scale 
after several crews had already been rated. The combined scores on each item can range from 2- 
14, so the regular composite scores can range from 8-56 (6-42 for videos) while the combined IP 
facilitation score can range from 24-168. 
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4. Results 

The research findings indicate that crew performance scores can be predicted from IP 
performance scores on the Debriefing Assessment Battery. The degree and quality of crew 
participation are highly correlated with several techniques utilized by IPs to facilitate discussion. 
Also, the findings that the obtained composite scores varied among IPs and crews, and that this 
variance is agreed upon between raters, suggest that the rating scale reliably measures differences 
in IP and crew performance. In this section, interrater reliabilities will be presented first, 
followed by data based on the comparison of IP and crew scores, and finally results of the 
correlations of IP and crew measures. Pearson product-moment correlation coefficients (r) were 
used for all correlation analyses. 

4.1 Interrater Reliability 

In order to test the interrater reliability of battery scores, Pearson correlation coefficients 
(r) were calculated. Scores initially assigned by both raters, prior to discussion and adjustments, 
were used for the calculation of the reliability coefficients. For the first five sessions from Airline 
Z, interrater reliability was moderate for overall IP scores (r = .60) and high for overall crew 
scores (r = .83). For the final four sessions reliability increased substantially for overall IP 
scores (r = .96) and remained high for overall crew scores (r = .81). For all 10 sessions 
combined, the reliabilities for overall IP (r = .82) and crew scores (r = .77) were both high. For 
the nine crews from Airline Y, interrater reliability was high for overall IP scores (r = .77) and 
moderate for overall crew scores (r = .53), though both were lower than the reliabilities for 
Airline Z. This decrease in reliability may be attributed to the fact that the sessions were longer 
and more complex for Airline Y. Finally, the reliabilities for all 19 sessions from Airlines Y and Z 
combined were high for both overall IP (r = .78) and overall crew scores (r = .73), as were the 
reliabilities for scores on each of the IP and crew measures (see Table 1). 

4.2 Comparisons of IP and Crew Scores 

In order to determine whether there are significant differences between the mean scores of 
the two airlines, 1-tests were calculated for the means of each IP and crew variable across airlines. 
It was determined that there are no significant differences in means, and therefore the scores from 
the two airlines will be combined for all remaining analyses (see Table 1). The only exception is 
videos, for which there is a statistically significant difference between the means of the two 
airlines, 1(17) = 2.25, p < .05. 
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Table 1 

Interrater Reliabilities, Mean Scores, and f-values Across Airlines 





Z Crews (n = 10) 

Y Crews (n = 9) 


Subjective 

variables 

N 

Interrater 

reliability 

Mean 

SO 

Mean 

SD 

i value 

IB 








Introduction 

18 

.92 

10.80 

3.26 

16.38 

10.35 

1.47 

Questions 

19 

.72 

33.20 

15.70 

40.22 

5.26 

1.33 

Encouragement 

19 

.79 

30.90 

15.37 

40.67 

5.29 

1.89 

Content 

19 

.82 

33.30 

13.98 

40.11 

5.46 

1.42 

Videos 

18 

.77 

33.30 

7.17 

25.25 

8.03 

2.25* 

Overall IP 

19 

.78 

141.50 

49.94 

162.63 

27.65 


Crew 








Content 

19 

.74 

33.20 

14.70 

38.00 

6.86 

.93 

Activity 

19 

.70 

35.30 

15.03 

40.56 

8.88 

.94 

Overall Crew 

19 

.73 

68.50 

29.40 

78.56 

15.04 ' 



*e< -05. 
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In order to determine the distribution of IP and crew scores on the assessment battery, 
frequencies were tabulated for each variable (see Table 2). While the majority of IPs scored poor 
on introductions (very few gave one), there was more variation among IPs and crews on the 
remaining variables, with scores averaging in the good range. 


Table 2 

Frequencies of Rating Scores on the Debriefing Assessment Battery 





Rating Scores (Average of the two raters) 


Subjective 



Needs 



Very 


variables 

Poor 

Marginal 

Improve 

Adequate 

Good 

Good 

Outstanding 

IP 








Introduction 

12 

4 

1 

0 

i 

0 

0 

Questions 

1 

2 

1 

4 

4 

7 

0 

Encouragement 

2 

2 

0 

2 

8 

5 

0 

Content 

1 

2 

0 

3 

9 

4 

0 

Videos 

0 

1 

3 

3 

2 

8 

1 

Crew 








Content 

1 

2 

1 

3 

8 

4 

0 

Activity 

1 

0 

5 

1 

5 

6 

1 
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4.3 Relationships Among IP and Crew Scores 

Relationships between subjective Debriefing Assessment Battery variables. 

In order to determine whether subjective ratings of crew activity and crew content can be 
predicted from subjective IP scores, Pearson correlation coefficients were calculated between all 
IP composite scores (predictor variables) and the two crew composite scores (criterion variables) 
on the Debriefing Assessment Battery (see Table 3). For each variable in the study the sample size 
is 19, except introduction and videos, for which N = 18 (the audio recording began after one session 
had already started, and the video equipment was non-functional for another session). An alpha 
level of .05 was used for all statistical tests in this study. 


Table 3 

Correlations Between Subjective IP Variables and Subjective Crew Variables 


Subjective Instructor variables 3 


Subjective 3 
Crew variables 

Introduction 

Questions 

Encourage 

Content 

Videos 

Activity 

-.17 

.54* 

.68 *** 

.57* 

-.14 

Content 

-.10 

.70 *** 

.77 *** 

.76 *** 

-.08 


3 See Debriefing Assessment Battery (Appendix B) 
*p < .05. **p<.01. ***p<.001. 
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Composite IP Encouragement Scores 









The correlations of crew activity with IP questions (r = .54), IP encouragement 
(r = .68), and IP content (r = .57) are all moderately high and statistically significant. The 
correlations of crew content with IP questions (r = .70), IP encouragement (r = .77), and IP 
content (r = .76) are high and are also statistically significant. The correlations of crew scores 
with IP introduction and video scores are not statistically significant (r = -.17 and -.14 
respectively for crew activity; r = -.10 and -.08 respectively for crew content). The similarity 
of the patterns of these IP and crew scores can be seen by comparing Figures 1 and 2, which present 
the actual encouragement scores for each IP from Airline Z and the actual content scores for each 
crew from airline Z. These figures demonstrate how closely the IP and crew scores correspond with 
each other across sessions.. 

To determine how closely related each subjective IP score is to the others, a correlation 
matrix was computed for intercorrelations between the subjective IP measures (see Table 4). 
Questions, encouragement, and content are all highly and significantly correlated with each other at 
the .001 level. This indicates that the effects of one variable cannot be separated from the others, 
so correlations of each of these variables with crew variables and objective IP variables may be 
indirectly affected by the others. 


Table 4 

Intercorrelations Between Subjective IP Variables 


Subscales 

1 

2 

3 

4 

5 

1 . Introduction 

- 

.35 

.30 

.40 

.15 

2. Questions 



.94 *** 

94 ... 

.16 

3. Encourage 



-- 

gg *** 

.03 

4. Content 




— 

.08 

5. Videos 





— 


*p < .05. **p<.01. ***p<.001. 
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Correlations between subjective variables from the Debriefing Assessment 
Battery and objective variables from study by Dismukes et al. (1995V Pearson 
correlations were calculated to determine the relationship between subjective scores on the 
Debriefing Assessment Battery and objective measures from the study by Dismukes et al. (1995). 
Definitions of the objective IP and crew measures are presented in Appendix C. 

First, correlations were calculated to determine the relationship between the subjective IP 
scores and five objective measures of crew participation (see Table 5). The only significant 
correlations are for the objective measures percent of crew participation (r = .57) and number of 
crew analyzing utterances per hour (r = .58) with the subjective measure of IP encouragement. 


Table 5 

Correlations Between Subjective IP Variables and Objective Crew Variables 


Subjective Instructor variables 3 


Objective 13 
Crew variables 

Introduction 

Questions 

Encourage 

Content 

Videos 

Percent participation 

-.19 

.34 

.57* 

.33 

-.38 

Words per Response 

-.11 

.27 

.31 

.39 

.12 

Self-initiated words 

-.29 

.11 

.36 

.09 

-.23 

Analyzing utterances 

-.28 

.38 

.58 ** 

.43 

-.20 

Proactive questions 

.00 

-.16 

-.01 

-.24 

-.26 

3 See Debriefing Assessment Battery (Appendix B) 

b See Appendix C 

*p < .05. **p<.01. ***p < .001 . 
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Next, correlations were calculated between subjective and objective crew variables to 
determine whether the two types of variables measure similar characteristics of crew 
participation. The correlations of the subjective measure of crew activity with the objective 
measures of percent participation (r = .86), number of words per response (r = .63), self- 
initiated statements (r = .65), and analyzing utterances per hour (r = .80) are all high and 
statistically significant (see Table 6). The correlations of the subjective measure of crew content 
with percent participation (r = .71 ), number of words per response (r = .65), and analyzing 
utterances per hour (r = .71) are moderately high and are also statistically significant. 


Table 6 

Correlations Between Subjective and Objective Crew Variables 


Objective variables 3 

Subjective 

variables 13 

Percent 

participation 

# of words 
per response 

Self-initiated 

words 

Analyzing 

utterances 

Proactive 

questions 

Activity 

.86 *** 

.63 ** 

.65 ** 

.80 *** 

.08 

Content 

y -j *** 

.65 ** 

.45 

.71 *** 

-.08 


a See Appendix C 

15 See Debriefing Assessment Battery (Appendix B) 
*p < .05. **p<.01. ***p<.001. 


Finally, correlations were calculated between subjective and objective IP measures to 
determine whether they measure similar components of IP facilitation. The three subjective IP 
measures of questions, encouragement, and content were found to have statistically significant 
correlations with three objective IP measures (see Table 7). The subjective measure of questions 
is moderately correlated with the objective measures of number of directed questions IPs ask per 
hour (r = .62) and the total number of questions IPs ask per hour (r = .57). IP encouragement is 
moderately correlated with percent of IP participation (r = -.58), number of directed questions 
asked per hour (r = .50), and total number of questions per hour (r = .47). IP content is 
correlated with number of directed questions asked per hour (r = .55). 
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Table 7 

Correlations Between Subjective and Objective IP Variables 


Objective variables 3 


Subjective 

variables b 

Percent 

participation 

# of words 
performance 

Directed 

questions 

Non-direct 

questions 

Total 

questions 

Introduction 

.18 

-.08 

.66 ** 

-.27 

.58 * 

Questions 

-.34 

.09 

.62 ** 

-.18 

.57* 

Encourage 

-.58 ** 

.04 

.50* 

-.12 

.47* 

Content 

-.34 

.21 

.55* 

-.28 

.45 

Video 

.38 

.38 

.25 

.01 

.26 


a See Appendix C 

b See Debriefing Assessment Battery (Appendix B) 
*p < .05. **p<.01. ***p<.001. 
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5. Conclusions and Recommendations 

The results of this study indicate that the ratings assigned to IPs on the measures of 
questions, encouragement, and content on the Debriefing Assessment Battery can be used to predict 
the degree and depth of crew participation. In this section the relationships that exist between IP 
and crew measures will be discussed, followed by the patterns of IP and crew scores obtained from 
the battery. Finally, basic recommendations for the development and implementation of future 
training programs will be presented. 

Of the five IP categories in the Debriefing Assessment Battery, three are highly and 
significantly related to the crew measures of activity and content. The measure of content of crew 
discussion is highly correlated with the IP measures of content, encouragement, and questions. This 
indicates that the depth to which crew members analyze the LOFT situation and evaluate their 
performance is directly related to the degree to which the IP facilitates and encourages them to do 
so. The fact that crew content is highly related to IP content also suggests that this component of IP 
performance on the Debriefing Assessment Battery does target techniques that tend to evoke in- 
depth analysis and evaluation by crews. 

The IP technique which best predicts crew activity is encouragement, followed by content 
and questions. This indicates that crew activity is directly related to the degree to which IPs 
refrain from instructing to the crew and encourage the crews to participate fully and initiate 
discussion. The fact that these IP measures can predict crew activity also suggests that the IP 
measures do capture techniques helpful in evoking active crew participation. 

On each of the above measures crew content is more highly related to the IP variables than is 
crew activity. Part of the reason crew activity is not as easily predicted by IP techniques as crew 
content may be that crews have their own distinct personalities. They tend to enter the sessions 
with varying propensities to talk. In some instances, no matter how hard the IP works to facilitate 
discussion the crew will not cooperate. At the other extreme, some crews may participate actively 
regardless of what the IP does or doesn't do to facilitate their participation. 

We cannot currently judge whether introductions can predict crew activity and content 
because not enough IPs gave introductions. While it is clearly possible to facilitate effectively 
without providing an introduction, theories of adult learning assert that introductions can enhance 
the quality of crew participation substantially because adults perform better when they know what 
is expected of them. It is therefore expected that if IPs begin giving thorough introductions in 
which they make it clear that they are facilitators and the crews are expected to analyze their own 
performance, crews will give more in-depth and thoughtful self-analyses. 

While videos are an important element of the debriefing because they provide the crew with 
the ability to view their performance objectively, the current measure of video usage is not a good 
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predictor of either crew content or activity. While videos might not be expected to predict crew 
activity, the learning literature assumes that video usage should enhance content of discussion. The 
problem may be that our measure is not sensitive enough to factors that could predict content. Our 
measure focuses simply on the effectiveness of IP use of the equipment and utilization of segments to 
introduce topics for discussion, and does not probe deeper aspects such as the content of the videos. 
Perhaps a more sensitive measure can be developed to better predict content. 

The fact that the subjective scores assigned on the Debriefing Assessment Battery are highly 
and significantly correlated with several objective measures of IP facilitation and crew 
participation suggests that the subjective rating scale is measuring valid aspects of IP and crew 
performance. To the extent they are not correlated the findings suggest that the subjective and 
objective measures are measuring differences as well as similarities. It is our belief that the 
subjective measures are identifying aspects of IP and crew performance that go beyond those 
captured by objective measures alone. The goal of developing the battery was to provide a tool for 
assessing IP facilitation skills and aspects of crew participation that are too subtle to be measured 
objectively. 

The frequencies of individual scores assigned on the Debriefing Assessment Battery indicate 
that on all of the categories except introductions IPs are doing a relatively good job of promoting 
crew participation, although there is still room for improvement. With instruction and practice 
IPs should be able to develop techniques to maximize active and in-depth evaluation and analysis by 
the crews The majority of IPs did an inadequate job of introducing the debriefing session and setting 
expectations for how the session would be conducted. Since the preponderance of adult learning 
literature asserts that it is essential to set such expectations, this is an area that needs 
improvement. Instructors should be made aware of the importance of providing an introduction and 
given examples of what constitutes an effective introduction. 

The purpose of the larger study of which the present study is a part is to determine what 
techniques IPs should use to most effectively facilitate crews in leading their own debriefings. In 
the present study it was found that ratings of crew participation can be predicted from ratings of IP 
facilitation. This indicates that what the IPs do has a significant impact on how well crews achieve 
the goal of active participation and in-depth analysis and evaluation of their LOFT performance; the 
more IPs work to encourage crew participation and probe for in-depth analysis, the higher crews 
rate on these measures. Thus, the above findings make a strong argument for the importance of 
providing IPs with clear guidelines and examples of good facilitation, and of providing training to 
IPs so they can develop their facilitation skills. Facilitation is not a skill that comes naturally; it 
must be practiced and developed to be implemented effectively. Currently, IPs are told that their 
objective is to facilitate crew self-analysis, but they are not given clear training or guidance on 
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how to do this. Therefore, the author recommends that the airlines develop and implement train- 
the-trainer programs to provide IPs with the skills and practice necessary to become effective 
facilitators. 

Some of the key elements that should be included in airline industry train-the-trainer 
programs can be derived from Wexley and Latham's (1991) training guidelines. They include (a) 
how to establish and communicate training objectives, (b) basic principles of how adults learn, so 
IPs can develop an understanding of what factors facilitate and interfere with learning, (c) how to 
communicate more effectively and use the LOFT videotape as a tool for facilitation, and (d) how to 
pose questions in a way that elicits active crew participation and discussion, including how to draw 
out quiet members. 

The final step in the development of any effective training program is to evaluate its 
effectiveness. Unfortunately it is currently impractical to track crews who participate in LOFT 
training to see if what they learn transfers to the line due to the strict standard of anonymity. As in 
the current study, the identities of all crew members, IPs, and airlines must remain confidential in 
order to preserve the learning capability of the LOFT. However, in order to determine the true 
effectiveness of the LOFT and facilitation techniques used for debriefing, longitudinal studies that 
track the progress and line performance of participants are needed (Gregorich & Wilhelm, 1993). 
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Appendix A 

Methods for Study by Dismukes et al. (1995) 

This study, which began in the summer of 1 994, is based on observations at a cross-section 
of major U.S. airlines. We have to date visited six companies. Our basic technique is to sit in on 
five to ten debriefings from a cross-section of the fleets at each company. For the most part we 
observe one debriefing per instructor and crew, though for about 20% of the instructors we have 
observed debriefing of a second crew for the purpose of comparison. For context, we interview 
managers in the CRM departments and review the written scenarios for each LOFT. We observe at 
least one LOFT simulation of each type in each fleet, but do not observe all of the LOFTs flown by all 
of the crews. 

We obtain permission from each instructor and each crew member to attend the debriefing 
and to audiotape the session. We provide assurance that all data collected will be completely de- 
identified to assure anonymity for subjects. After the debriefing, we interview the instructor, ask 
him or her to rate the crew's CRM performance and their technical proficiency on a 5 point Likert 
scale, and ask for comments about the debriefing process. 

The audio recordings of the debriefings are later transcribed into text in their entirety and 
all references to individuals and organizations are deleted. Each utterance is coded for each of the 
factors in Table 1 . We also record how many video segments from the LOFT were played, how long 
each lasted, and how each segment is discussed in the debriefing. 
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Table 1. Coding of Utterance Factors 

Utterance length: 

number of words 

Speaker: 

Instructor (IP), Captain (CA), First Officer (FO), or Flight 
Engineer (FE) 

Target of Question (if directed 

Instructor (IP), Captain (CA), First Officer (FO), or Flight 

to a particular crew member): 

Engineer (FE) 

Utterance type: 

Question, Command, Response, or Statement (SI for 
statements self-initiated by crew) 

Topic type: 

CRM, Technical, Mixed, or Non-Specific 

Topic category: 

Introduction or Summary, Scenario, Static (general 
knowledge), crew performance (categorized as Positive, 
Negative, how to Improve, or Neutral description) 

CRM subtopic: 

e.g., workload, communication, team building 

Technical subtopic: 

e.g., abnormal procedures, aircraft systems, weather 
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Appendix B 


DEBRIEFING ASSESSMENT BATTERY 


INSTRUCTOR PROFILE 

The Instructor Profile is a summary of the strategies and techniques IP's use to guide the debriefing 
session. The two main goals of the debriefing session are to 1) get the crew to perform an in-depth 
analysis of the situation that confronted them, how they understood and managed the situation, the 
outcome, and ways to improve, and 2) get the crew to participate in a proactive, rather than reactive, 
manner in which they initiate discussion and elaborate beyond the minimal. 


Directions: 

Use the scale below to rate the instructors on each of the following elements, then total the scores to get 
the overall rating for each category 

Needs 

Poor Marginal Improvement Adequate Good Very Good Outstanding 

1 2 3 4 5 6 7 


INTRODUCTION 

One purpose of the introduction is to let the crew know that participation and self-evaluation are expected 
of them, and why it is important. 

Makes clear that his role is guide/facilitator and that crew should do most of the talking 

Clearly conveys that crew should take an active role, initiating discussion rather than just 

responding to him 

Clearly conveys that he wants crew to dig deep, critically analyzing the LOFT and their 

performance 

Gives a persuasive rationale for the crew to participate actively and make their own analysis 

Overall rating of introduction 


QUESTIONS 

The purpose of asking questions is to get the crew to participate, focus the discussion on important topics, 
and enlist the crew in discussing the topics in depth. 

Asks an appropriate number of questions to get crew talking & lead them to issues 

Avoids answering for the crew when they do not respond immediately or correctly and uses a 

pattern of questioning that keeps the focus on the crew 

Uses probing and follow-up questions to get crew to analyze in depth and to go beyond yes/no 

and brief factual answers 

Uses questioning techniques to encourage interaction and sharing of perspectives among crew 

members 

Overall rating of questions 
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ENCOURAGEMENT 

Encouragement refers to the degree to which the instructor encourages and enables the crew to actively 
and deeply participate in the debriefing. 

Conveys sense of interest in crew views and works to get them to do most of the talking 

Encourages continued discussion through active listening, strategic pauses, avoiding disruptive 

interruptions, and/or following up on crew-initiated topics 

Encourages all members to participate fully, drawing out quiet members if necessary 

Refrains from giving long soliloquies or giving his own analysis before crew has fully analyzed 

Overall rating of encouragement 


CONTENT OF DISCUSSION 

The goal of the debriefing session is to get the crew to evaluate and analyze their own CRM performance so 
they will leam more deeply and can gain practice in debriefing themselves, a skill they can then begin to 
use on the line. 

Encourages crew to analyze along CRM dimensions the situation that confronted them, what 

they did to manage the situation, and why they did it 

Encourages crew to evaluate their performance and/or ways they might improve 

Encourages crew to explore CRM issues and how they specifically affect LOFT performance and 

line operations 

Encourages crew to analyze issues, factors, and outcomes in depth, going beyond simply 

describing what happened and what they did 

Overall rating of content of discussion 


VIDEOS 

One stated purpose of showing videotaped segments of the LOFT is to enable the crew members to see how 
they performed from an objective viewpoint so they can better evaluate their performance. More 
realistically, perhaps, the video reminds the crew of the situation, aiding their memory and providing a 
focus for discussion. 

Shows an appropriate number of videos of appropriate duration to illustrate/introduce topics 

Uses video equipment efficiently: is able to find desired segment without wasting time and 

pauses the video if substantial talk begins while playing 

Consistently discusses video segments, using them as a springboard for discussion of specific 

topics 

Has a point to make and uses the video to make that point. 

Overall rating of video usage 


.Overall instructor rating 
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CREW PROFILE 

The crew profile measures the degree and depth of participation by the crew. 


Directions: 

Use the scale below to rate the crew on each of the following elements, then total the scores to get the 
overall rating for each category 


Needs 

Poor Marginal Improvement Adequate Good Very Good Outstanding 

1 2 3 4 5 6 7 


CONTENT OF DISCUSSION 

Content of analysis refers to the depth to which the crew members analyze the LOFT situation and their 
performance. 

Analyze along CRM dimensions the situation that confronted them, what they did to manage the 

situation, and why they did it 

Evaluate their performance and ways they might improve 

Explore CRM issues and how they affect LOFT performance and line operations 

Analyze issues, factors, and outcomes in depth, going beyond simply describing what happened 

and what they did 

Overall rating of content of discussion 


ACTIVITY 

Activity refers to how actively, versus passively, the crew participates in and initiates discussion. 

Go beyond minimal responses to IP questions 

Participate deeply and thoughtfully 

Initiate dialogue rather than just responding to questions, and/or interact with each other 

rather than only with the IP 

Behave in a predominantly proactive rather than reactive manner, being actively involved 

rather than just passing through the training 

Overall rating of activity 


.Overall crew rating 
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Appendix C 

Objective Measures 


OBJECTIVE IP VARIABLES 


Percent Participation: percentage of total words during debriefing sessions spoken by IPs 

Number Of Words Performance: average number of IP words relating to crew performance 

Directed Questions: average number of questions IPs asked per hour that were directed to 
specific crew members 

Non-Directed Questions: average number of questions IPs asked per hour that were not 
directed to specific crew members 

Total Questions: average number of total questions asked by IPs per hour 


OBJECTIVE CREW VARIABLES 


Percent Participation: percentage of total words during debriefing sessions spoken by crew 
members 

Number Of Words Per Response: average number of words per crew member response to IP 
questions 

Self-Initiated: average number of words per hour initiated by crew members 

Analyzing Utterances: average number of analyzing utterances per hour by crew members 

Proactive Questions: average number of questions initiated by crew members that are not 
responsive to the IP or miscellaneous in content 
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